from alarms import *
print(pio.renderers)
# pio.renderers.default = "vscode"
# pio.renderers.default = "notebook"

Renderers configuration
-----------------------
    Default renderer: 'plotly_mimetype+notebook'
    Available renderers:
        ['plotly_mimetype', 'jupyterlab', 'nteract', 'vscode',
         'notebook', 'notebook_connected', 'kaggle', 'azure', 'colab',
         'cocalc', 'databricks', 'json', 'png', 'jpeg', 'jpg', 'svg',
         'pdf', 'browser', 'firefox', 'chrome', 'chromium', 'iframe',
         'iframe_connected', 'sphinx_gallery']

path = "./data/stats-with-IOPs.csv"
df = pd.read_csv(path, low_memory=False)
df["StartTime"] = df["StartTime"].apply(lambda d: parse(d))
df["EndTime"] = df["EndTime"].apply(lambda d: parse(d))
df["Time"] = df["Time"].apply(lambda d: datetime.strptime(d,"%H:%M").time())
# for col in df.columns:
#     print(col, type(df[col][0]))
# print("===============")
# df.info()
df.head(20)

# df.describe()

StartTime¶

The following graph shows all the alarms triggered in this dataset. The x-axis represents the activation time of an alarm, and the y-axis shows the duration (i.e., TimeDelta= StartTime - EndTime) of the corresponding activation. As we can see that most of the alarms deactivated within 20 seconds (it will be more clear in followings sections).

fig = px.scatter(df, x="StartTime", y="TimeDelta",render_mode="webgl")
fig.show()

Box Plot of TimeDelta for all alarms¶

This is one of the most important box plots in the whole analysis. It will help us to determine the threshold for TimeDelta. If the duration between activation and deactivation (i.e., TimeDelta = Deactivation - Activation) is less than the threshold then we will not transmit such alarm to the historian. From the following box plot, we can see that the first quartile (q1) is equal to 16 seconds which means if we set the threshold constant equals to 16 seconds we will directly reduce 25% of the communications between the DCS systems and Historian server. Similarly, if we set threshold constant to 29 seconds (i.e., q2 or median value) the 50 % of the storage & and communication will be reduced.
Questions:

We need more data to determine the threshold constant value. Can I get more data of the same plant? For example, 1 year or 6 months.
Can I choose threshold value equals to 29 seconds?

Note: If hover the mouse over any graph it will show the values.

fig = px.box(df, y="TimeDelta")
fig.update_yaxes(range=[0, 100])
fig.show()

SourceName Analysis¶

From the first histograms, we can see that "47TI931" triggered the most number of alarms. Additionally, all the alarms are related to “IOP” condition.

Questions

Why so many communications problems occur? Is it normal?

fig = px.histogram(df, x = "SourceName")
fig.update_layout(title="Number of times each SourceName triggered an alarm.")
fig.show()

fig = px.histogram(df, x = "SourceName", color="Condition",  barmode='group')
fig.update_layout(title="Number of times a condition occured for a SourceName.")
fig.show()

fig = px.box(df, x= "SourceName", y="TimeDelta")
fig.update_yaxes(range=[0, 200])
fig.update_layout(title="Box Plot of TimeDelats (Deactivaion - Activation) for each SourceName")
fig.show()

Conditions Analysis¶

As we can see from the following graphs, the most frequent alarms are related to communication problems. For instance, IOP condition occurred more 191 thousand time. Additionally, from the box plot, we can see that that durations (i.e., timedelta = deactivation time - activation time) of IOP alarms are shorter.

Conclusion: IOP communications can discard easily as it just represent only the communication issue between the field device and the DCS system. So if we discard such alarms we reduce the communication between the DCS system and historian server to a few hundreds instead of thousands.

Questions:

Why the Vel+ and Vel- conditions are lesser than compared to other conditions? Because they were the most in 1-day data (old data). Is it normal?
Whether my conclusion is correct or not?

fig = px.histogram(df, x = "Condition")
fig.show()


fig = px.box(df, x= "Condition", y="TimeDelta")
fig.update_yaxes(range=[0, 200])
fig.show()

Condtions Occuring on each day of Week¶

fig = px.histogram(df, x="WeekDay", color="Condition",  barmode='group')
fig.show()

sources = {} # contains each unique sourcename as key and corresponding alarms in a list
for sname in df["SourceName"].unique():
    sources[sname] =  []
temp_dict = None
for i in range(df.shape[0]):
    temp_dict = {}
    for key in df.columns:
        temp_dict[key] = df[key][i]
    sources[df["SourceName"][i]].append(temp_dict)

Chattering¶

If an alarm from the same source is triggered 3 times or less then such situation of chattering is considered normal. However, from the following the histograms, we can see that thousands of alarms are triggered more than thrice in most of the sensors. For instance, consider "47TI931A", it chatters 6 times in a minute more than 31 thousand times.

Note: x-axis represents the number of times an alarm chatter in the duration of one minute. The y-axis represents how many times such conditions (i.e., chatters) occur.

Conclusion: Such conditions (in which an alarm is triggered more than 3 times in a minute) are abnormals. So, if we detect such conditions at the edge it may help to do.... [list the usages].

Questions:

Is my conclusion correct?
What are the use cases if we detect chatters in real-time?
How too much chattering effect system performance?

cols = 2
rows = math.ceil(len(sources.keys())/cols)
fig = make_subplots(rows=rows, cols=cols)


t = 0
snames = list(sources.keys())
for row in range(1, rows+1):
    for col in range(1,cols+1):
        chats = findChatterings(sources[snames[t]])
        chats = [v["count"] for v in chats.values()]
        trace = go.Histogram(x=chats,name=snames[t])
        fig.add_trace(trace,row =row, col=col)
        t += 1
fig.show()

Time Between form one Deactivation to Next Activation for whole Data¶

In this section, I tried to find the time between 1 deactivation to next activation. In ideal scenario, there should be 1 alarm per minute. However, when I took the time difference between 1 deactivation to the next activation of any alarm the difference (delta) was mostly under 10 seconds which means that a huge number alarms are generated in 1 minute.

Note: The x-axis represents the time delta (i.e., the time difference between deactivation and activation of an alarm), while the y-axis represents the frequency (count) of corresponding deltas.

Questions:

Are my observations correct? If not please let me know what should I do to cross-check my results
If my observations are correct then what kind of conclusions I can draw from this graph?

alarms = []
# concatenating lists of alarms
for sname in sources.keys():
    alarms = alarms + sources[sname] 

durations = frequencyOfAlarmsActivated(alarms)
d = {}
for v in durations:
#     if v < 200:
    d[v] = 0

for v in durations:
#     if v < 200:
    d[v] +=1

counts = [v for v in d.values()]
deltas = [k for k in d.keys()]

fig = px.bar(x=deltas, y = counts)
fig.show()

for delta in sorted(d.keys()):
    print((delta, d[delta]), end=" ")

(0.0, 41699) (1.0, 43862) (2.0, 128638) (3.0, 26311) (4.0, 21472) (5.0, 8144) (6.0, 5180) (7.0, 3056) (8.0, 2166) (9.0, 1349) (10.0, 837) (11.0, 575) (12.0, 406) (13.0, 265) (14.0, 194) (15.0, 128) (16.0, 87) (17.0, 75) (18.0, 75) (19.0, 33) (20.0, 50) (21.0, 22) (22.0, 31) (23.0, 10) (24.0, 15) (25.0, 9) (26.0, 12) (27.0, 8) (28.0, 3) (29.0, 5) (30.0, 6) (32.0, 2) (33.0, 1) (34.0, 6) (35.0, 4) (36.0, 2) (37.0, 2) (43.0, 2) (1411.0, 2) (63734.0, 1)

	SourceName	StartTime	EndTime	Message	RecoveryMessage	Condition	Mask	NewState	Status	TimeDelta	Year	Month	MonthDay	WeekDay	Time	Hour	Minute
0	47TI931A	2019-03-06 13:19:17	2019-03-06 13:19:33	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	16.0	2019	3	6	Wednesday	13:19:00	13	19
1	47TI931A	2019-03-06 13:19:35	2019-03-06 13:20:22	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	47.0	2019	3	6	Wednesday	13:19:00	13	19
2	47TI931A	2019-03-06 13:20:24	2019-03-06 13:20:28	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	4.0	2019	3	6	Wednesday	13:20:00	13	20
3	47TI931A	2019-03-06 13:20:30	2019-03-06 13:20:49	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	19.0	2019	3	6	Wednesday	13:20:00	13	20
4	47TI931A	2019-03-06 13:20:51	2019-03-06 13:21:03	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	12.0	2019	3	6	Wednesday	13:20:00	13	20
5	47TI931A	2019-03-06 13:21:05	2019-03-06 13:21:11	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	6.0	2019	3	6	Wednesday	13:21:00	13	21
6	47TI931A	2019-03-06 13:21:13	2019-03-06 13:21:25	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	12.0	2019	3	6	Wednesday	13:21:00	13	21
7	47TI931A	2019-03-06 13:21:27	2019-03-06 13:22:22	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	55.0	2019	3	6	Wednesday	13:21:00	13	21
8	47TI931A	2019-03-06 13:22:25	2019-03-06 13:23:51	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	86.0	2019	3	6	Wednesday	13:22:00	13	22
9	47TI931A	2019-03-06 13:23:53	2019-03-06 13:24:33	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	40.0	2019	3	6	Wednesday	13:23:00	13	23
10	47TI931A	2019-03-06 13:24:35	2019-03-06 13:24:40	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	5.0	2019	3	6	Wednesday	13:24:00	13	24
11	47TI931A	2019-03-06 13:24:42	2019-03-06 13:25:10	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	28.0	2019	3	6	Wednesday	13:24:00	13	24
12	47TI931A	2019-03-06 13:25:12	2019-03-06 13:25:44	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	32.0	2019	3	6	Wednesday	13:25:00	13	25
13	47TI931A	2019-03-06 13:25:46	2019-03-06 13:25:48	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	2.0	2019	3	6	Wednesday	13:25:00	13	25
14	47TI931A	2019-03-06 13:25:50	2019-03-06 13:25:56	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	6.0	2019	3	6	Wednesday	13:25:00	13	25
15	47TI931A	2019-03-06 13:25:58	2019-03-06 13:26:00	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	2.0	2019	3	6	Wednesday	13:25:00	13	25
16	47TI931A	2019-03-06 13:26:02	2019-03-06 13:27:06	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	64.0	2019	3	6	Wednesday	13:26:00	13	26
17	47TI931A	2019-03-06 13:27:08	2019-03-06 13:27:51	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	43.0	2019	3	6	Wednesday	13:27:00	13	27
18	47TI931A	2019-03-06 13:27:53	2019-03-06 13:28:16	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	23.0	2019	3	6	Wednesday	13:27:00	13	27
19	47TI931A	2019-03-06 13:28:18	2019-03-06 13:28:20	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C IOP	47TI931A C5 1.YTK GIRDISCAP SICAK PV = 691 C I...	IOP	201	3	1	2.0	2019	3	6	Wednesday	13:28:00	13	28

StartTime¶

Box Plot of TimeDelta for all alarms¶

SourceName Analysis¶

Conditions Analysis¶

Condtions Occuring on each day of Week¶

Chattering¶

Time Between form one Deactivation to Next Activation for whole Data¶

End¶